Image processing device, image procssing method, and computer program product

ABSTRACT

According to an embodiment, an image processing device includes a generator and a processor. The generator is configured to generate, from a plurality of unit images in which points on an object are imaged by an imaging unit at different positions according to distances between the imaging unit and the positions of the points on the object, a refocused image focused at a predetermined distance. The processor is configured to perform blurring processing on each pixel of the refocused image according to an intensity corresponding to a focusing degree of the pixel of the refocused image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2012-155834, filed on Jul. 11, 2012; theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an image processingdevice, an image processing method, and a computer program product.

BACKGROUND

There are known a light-field camera including a microlens array, cameraarray, or the like that simultaneously captures a plurality of images ofthe same object. In each of the images, a slightly different part of thesame object, which is slightly shifted from each other, is shown. Thus,it is possible to reconstruct an image focused at any distancedesignated for the image by shifting and superimposing the plurality ofimages. Such reconstructed image is referred to as a refocused image.

In the conventional technology, however, there is a problem thatvariation in pixel values between the superimposed images is large in aregion in which non-focused object is imaged, and thus, artifact of afalse image shape occurs in the refocused image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of the configurationof an image processing device according to a first embodiment;

FIG. 2 is a diagram schematically illustrating an example of theconfiguration of an acquisition unit according to the first embodiment;

FIG. 3 is a schematic diagram illustrating an example of an imageacquired by the acquisition unit according to the first embodiment;

FIG. 4 is a flowchart illustrating an example of an operation performedby the image processing device according to the first embodiment;

FIG. 5 is a flowchart illustrating an example of an operation ofgenerating a refocused image according to the first embodiment;

FIG. 6 is a schematic diagram illustrating an expanded form of eachmicrolens image according to the first embodiment;

FIG. 7 is a diagram illustrating generation of the refocused imageaccording to the first embodiment;

FIG. 8 is a flowchart illustrating an example of an operation ofcalculating the degree of non-focus according to the first embodiment;

FIG. 9 is a block diagram illustrating another example of theconfiguration of the image processing device according to the firstembodiment;

FIG. 10 is a flowchart illustrating an example of blurring processingaccording to the first embodiment;

FIG. 11 is a diagram illustrating an example in which upper and lowerlimit values are set for the size of a kernel according to the firstembodiment;

FIG. 12 is a diagram illustrating an example of a map of the degree ofnon-focus according to the first embodiment;

FIG. 13 is a block diagram illustrating an example of the configurationof an imaging device according to a second embodiment;

FIG. 14 is a block diagram illustrating an example of the configurationof a sensor device according to a third embodiment;

FIG. 15 is a block diagram illustrating an example of the configurationof an image processing system according to a fourth embodiment; and

FIG. 16 is a block diagram illustrating an example of the configurationof a computer device to which the image processing device is applicableaccording to another embodiment.

DETAILED DESCRIPTION

According to an embodiment, an image processing device includes agenerator and a processor. The generator is configured to generate, froma plurality of unit images in which points on an object are imaged by animaging unit at different positions according to distances between theimaging unit and the positions of the points on the object, a refocusedimage focused at a predetermined distance. The processor is configuredto perform blurring processing on each pixel of the refocused imageaccording to an intensity corresponding to a focusing degree of thepixel of the refocused image.

First Embodiment

Hereinafter, an image processing device according to the firstembodiment will be described. FIG. 1 is a diagram illustrating anexample of the configuration of an image processing device 100 accordingto the first embodiment. The image processing device 100 performs aprocess on an image acquired by an acquisition unit 101 according to thefirst embodiment. The image processing device 100 includes a refocusedimage generator 102, a non-focus degree calculator 103, and a blurringprocessor 104. The refocused image generator 102, the non-focus degreecalculator 103, and the blurring processor 104 may be configured bycooperative hardware, or some or all thereof may be configured by aprogram operating on a CPU (Central Processing Unit).

The acquisition unit 101 acquires a plurality of unit images in whichpoints on an object are imaged at different positions according todistances between the acquisition unit 101 and the positions of thepoints on the object.

FIG. 2 is a diagram schematically illustrating an example of theconfiguration of the acquisition unit 101 according to the firstembodiment. In the example of FIG. 2, the acquisition unit 101 includesa main lens 110 that serves as an imaging optical system and imageslight from an object 120 and a microlens array 111 in which a pluralityof microlenses are arrayed at a predetermined pitch. The acquisitionunit 101 further includes a sensor 112 that converts the light imaged bythe respective microlenses of the microlens array 111 into an electricsignal and outputs the electric signal. In the example of FIG. 2, themain lens 110 is set such that an image formation surface of the mainlens 110 is located between the main lens 110 and the microlens array111 (in an image surface Z).

Although not illustrated, the acquisition unit 101 further includes asensor driving unit that drives a sensor. Driving of the sensor drivingunit is controlled according to a control signal from the outside.

The sensor 112 converts the light received on the light receptionsurface into an electric signal and outputs the electric signal. Forexample, a CCD (Charge Coupled Device) image sensor or a CMOS(Complementary Metal Oxide Semiconductor) image sensor can be used asthe sensor 112. In such an image sensor, light-receiving elementscorresponding to respective pixels are configured to be arrayed in amatrix form on the light reception surface. Thus, the light is convertedinto the electric signal of each pixel through photoelectric conversionof each light-receiving element and the electric signal is output.

The acquisition unit 101 causes the sensor 112 to receive light incidenton a position on the microlens array 111 from a given position on themain lens 110 and outputs an image signal including a pixel signal ofeach pixel. An imaging device having the same configuration as theacquisition unit 101 is known as the name of a light-field camera or aplenoptic camera.

FIG. 3 is a diagram schematically illustrating an example of an imagecaptured and acquired by the acquisition unit 101. The acquisition unit101 acquires an image 130 in which images 131, 131, . . . formed on thelight reception surface of the sensor 112 by the respective microlensesof the microlens array 111 are arrayed in correspondence with the arrayof the microlenses. The image 130 in which the images 131 are disposedaccording to the array of the respective microlenses of the microlensarray 111 is referred to as a compound eye image 130 below. The images131, 131, are unit images which serve as units forming the compound eyeimage 130.

The images 131 formed by each microlens are preferably formed on thesensor 112 without overlap. Since each image 131 in the compound eyeimage 130 captured by the optical system exemplified in FIG. 3 is a realimage, an image obtained by extracting each image 131 and inversing theextracted image 131 to the right, left, upper, and lower sides isreferred to as a microlens image 131. The description will be made belowusing the microlens image as the unit image. That is, an image formed byone microlens is the microlens image 131, and an image in which theplurality of microlens images 131 are arrayed is the compound eye image130.

In the configuration of FIG. 2, the light from the object 120 is imagedas the respective microlens images 131 such that the entire or partialregion of the object 120 is slightly shifted according to the positionsof the respective microlenses. That is, the acquisition unit 101acquires two or more microlens images 131 imaged in a state in which thepoints of interest on the object 120 imaged commonly by two or moremicrolenses are shifted according to distances up to the respectivepoints of interest of the two or more microlenses. In other words, theacquisition unit 101 acquires the plurality of microlens images 131 inwhich the points of interest are imaged at different positions accordingto the distances from the plurality of microlenses.

The example has been described above in which the image formationsurface of the main lens 110 in the acquisition unit 101 is locatedbetween the main lens 110 and the microlens array 111, but theembodiment is not limited to this example. For example, the imageformation surface of the main lens 110 may be set on the microlens array111 or may be set located on the rear side of the sensor 112. When imageformation surface of the main lens 110 is located on the rear side ofthe sensor 112, the microlens image 131 formed in the sensor 112 is avirtual image.

The example has been described above in which the acquisition unit 101uses the microlens array 111 in which the plurality of microlenses arearrayed, but the invention is not limited to this example. For example,the acquisition unit 101 may use a camera array in which a plurality ofcameras are arrayed. When the microlens array is used, one sensor isgenerally used. On the other hand, when the camera array is used, eachof the cameras forming the camera array includes one sensor. A shrinkageratio of an image by the microlens corresponds to a shrinkage ratio of alens of the camera of the camera array. A pitch between the microlensescorresponds to a distance between the cameras of the camera array. Whenthe camera array is used, an image captured by each camera is a unitimage and an output image of the entire camera array is a compound eyeimage.

Referring back to FIG. 1, the refocused image generator 102 generates,from the compound eye image 130 acquired by the acquisition unit 101, arefocused image focused on the object at a designated distance from theacquisition unit 101 (the main lens 110). The blurring processor 104performs blurring processing on the refocused image output from therefocused image generator 102 with a blur intensity corresponding to thedegree of focus of each pixel of the refocused image. The refocusedimage subjected to the blurring processing by the blurring processor 104is output as an output image from the image processing device 100.

The blurring processing performed by the blurring processor 104 isperformed more specifically as follows. The non-focus degree calculator103 calculates, from the compound eye image 130 acquired by theacquisition unit 101, the degree of non-focus indicating the degree ofdefocus for each pixel of the refocused image. In this case, the degreeof non-focus is calculated so as to indicate a larger value when focusis further deviated (the degree of focus is lower). The blurringprocessor 104 performs the above-described blurring processing on therefocused image output from the refocused image generator 102 based onthe degree of non-focus. The refocused image subjected to the blurringprocessing by the blurring processor 104 is output as an output imagefrom the image processing device 100.

FIG. 4 is a flowchart illustrating an example of an operation performedby the image processing device 100 according to the first embodiment.First, in step S21, the acquisition unit 101 acquires the compound eyeimage 130. The acquired compound eye image 130 is supplied to therefocused image generator 102 and the non-focus degree calculator 103.Next, in step S22, the refocused image generator 102 generates, from thecompound eye image 130 supplied from the acquisition unit 101 in stepS21, the refocused image in which a focus position is changed. Next, instep S23, the non-focus degree calculator 103 calculates, from thecompound eye image 130 supplied from the acquisition unit 101 in stepS21, the degree of non-focus for each pixel of the refocused image. Instep S24, the blurring processor 104 performs the blurring processing onthe refocused image generated in step S22 based on the degree ofnon-focus calculated in step S23.

The operation of generating the refocused image in step S22 will bedescribed. In step S22, the refocused image generator 102 generates,from the compound eye image 130 supplied from the acquisition unit 101,the refocused image focused at a predetermined distance along thedirection from the acquisition unit 101 (the main lens 110) to theobject 120. The predetermined distance may be a distance determined inadvance or may be designated by a user through a user's input or thelike on an input unit (not illustrated). The refocused image generator102 generates the refocused image by expanding and superimposing theunit images at an expansion magnification corresponding to the distanceto be focused at.

FIG. 5 is a flowchart illustrating an example of an operation ofgenerating the refocused image performed by the refocused imagegenerator 102. Before the description of FIG. 5, a relation between thea distance to be focused at and an expansion magnification of themicrolens image 131 which is the unit image will be described withreference to FIG. 2. The image surface Z indicates an image surface ofan image to be generated through a refocus operation. A distance Aindicates a distance between the object 120 desired to be focused on andthe main lens 110. A distance B indicates a distance between the mainlens 110 and the image surface Z. A distance C indicates a distancebetween the image surface Z and the microlens array 111. A distance Dindicates a distance between the microlens array 111 and the sensor 112.An image of the object 120 for which a distance from the main lens 110is the distance A is assumed to be formed on the image surface Z.

An image on the image surface Z may be generated by expanding themicrolens images 131 C/D times, and superimposing them while beingshifted by an amount corresponding to the pitch between the microlenses.At this time, the distance A and the distance B have a one-to-onecorrespondence relation from a property of a lens. Therefore, when “thedistance B+the distance C” is set to a fixed distance K, the distance Aand the distance C have a one-to-one correspondence relation. Thus, byperforming inverse operation from the distance A so as to determine thevalue of the distance C, an expansion magnification m of the microlensimage 131 can be determined.

Referring back to the flowchart of FIG. 5, the description will be made.In step S91, the refocused image generator 102 calculates the expansionmagnification m determined by the distance A to be focused at, asdescribed above.

Next, in step S92, the refocused image generator 102 expands eachmicrolens image 131 at the expansion magnification m calculated in stepS91. A method of expanding the microlens image 131 is not particularlylimited. For example, a general image expansion method such as bilinearinterpolation or a cubic convolution interpolation method can beapplied. An image obtained by expanding an i-th microlens image 131 isassumed to be a microlens image M_(i).

FIG. 6 is a diagram schematically illustrating a case in which eachmicrolens image 131 is expanded at the expansion magnification m. In theexample of FIG. 6, microlens images 131 ₁, 131 ₂, and 131 ₃ captured bythree adjacent microlenses are each expanded at the expansionmagnification m. Superimposed portions are produced between expandedmicrolens images M₁, M₂, and M₃.

Next, in step S93, the refocused image generator 102 superimposes, foreach pixel, the microlens images M_(i) expanded in step S92 by shiftingthe microlens images M_(i) by a difference between the centralcoordinates o_(i) of the microlenses that form the microlens imagesM_(i). Next, in step S94, the refocused image generator 102 calculatesthe average pixel value of the pixels whose superimposed positionsaccord with each other, and then generates a refocused image 140 (seeFIG. 6) by using those average pixel value as a pixel value thereof.

The pixel value of a refocused image I can be calculated using thecentral coordinates o_(i) of the i-th microlens image M_(i) on thesensor 112 by using Equation (1) below. In Equation (1), it is assumedthat W(x) is a set of the numbers i of the microlens images M_(i)superimposed at coordinates x and a sign |·| indicates the number ofelements of the set.

$\begin{matrix}{{I(x)} = {\frac{1}{{W(x)}}{\sum\limits_{i \in {W{(x)}}}^{\;}\; {M_{i}\left( {x - o_{i}} \right)}}}} & (1)\end{matrix}$

The generation of the refocused image 140 will be described in detailwith reference to FIG. 7. For example, as exemplified in FIG. 7, thesuperimposed portions of the expanded microlens images M₁, M₂, and M₃,which are disposed about the central coordinates o₁, o₂, and o₃ of themicrolens images 131 ₁, 131 ₂, and 131 ₃ on the sensor 112 (see FIG. 6),respectively, are produced on the image surface Z.

In this case, for example, a pixel value at coordinates x₁ of asuperimposed portion of the expanded microlens images M₁ and M₂ can becalculated by averaging the pixel values positioned at the coordinatesx₁ when the expanded microlens images M₁ and M₂ are superimposed.Likewise, a pixel value at coordinates x₂ of a superimposed portion ofthe expanded microlens images M₁, M₂, and M₃ can be calculated byaveraging corresponding pixel values in the expanded microlens imagesM₁, M₂, and M₃.

The size of the refocused image 140 can be freely set. When the size ofthe refocused image 140 is set to 1/α of the resolution of the sensor112, the refocused image generator 102 may perform the refocusprocessing using a value obtained by multiplying the expansionmagnification by 1/α and a value obtained by each central coordinateso_(i) by 1/α.

Alternatively, the refocused image 140 may be generated by changing anamount of shift of the unit image without expanding each microlens image131. At this case, the refocused image 140 may be generated by Equation(1) using, as the amount of shift, a value obtained by multiplying eachcentral coordinates o_(i) by l/m based on the above-described expansionmagnification m.

In the case where the distance A is set later by a user, the expansionmagnification m may be directly set instead of setting the distance A.In this case, the user can repeatedly change the setting of theexpansion magnification m until the object to be focused on is focusedon, while checking the refocused image 140 according to the setexpansion magnification m.

Even when the compound eye image captured by the camera array isprocessed, the refocused image 140 can be generated by calculating anaverage pixel value of superimposed pixels obtained by shifting andsuperimposing the unit images using, as the amount of shift, an amountof parallax between the cameras which is determined according to thefocused distance A.

The operation of calculating the degree of non-focus in step S23 of theflowchart of FIG. 4 will be described. The non-focus degree calculator103 calculates the degree of non-focus indicating lowness of the degreeof focus for each pixel of the refocused image 140. More specifically, avalue of the degree of non-focus is larger, as the focus is lessachieved, that is, the degree of focus is lower.

The non-focus degree calculator 103 calculates a variation in the pixelvalues of the pixels whose positions accord with each other in thesuperimposed portions where the expanded microlens images M_(i) expandedfrom the unit images (microlens images 131) are superimposed. Then, thenon-focus degree calculator 103 calculates the degree of non-focus suchthat the value of the degree of non-focus is larger as the variation inthe calculated pixel values is larger.

FIG. 8 is a flowchart illustrating an example of an operation ofcalculating the degree of non-focus J performed in the non-focus degreecalculator 103. In step S1001 to step S1003, the non-focus degreecalculator 103 expands the microlens images 131 using the same expansionmagnification m as that of the above-described process performed by therefocused image generator 102 and the central coordinates o_(i) of themicrolenses and superimposes the expanded microlens images M_(i). Sincethe processing from step S1001 to step S1003 is the same as theprocessing from step S91 to step S93 performed by the refocused imagegenerator 102, as described with reference to FIG. 5, the detaileddescription thereof will not be repeated here.

After the non-focus degree calculator 103 superimposes the microlensimages Mi, in step S1004, the non-focus degree calculator 103subsequently calculates a variation in the pixel values of the pixelswhose positions accord with each other when superimposed. For example,in regard to the variation, a variance of the pixel values of the pixelsin the microlens images Mi whose positions accord with each other whensuperimposed is calculated as the degree of non-focus J by Equation (2).

$\begin{matrix}{{J(x)} = {\frac{1}{{W(x)}}{\sum\limits_{i \in {W{(x)}}}^{\;}\; \left( {{M_{i}\left( {x - o_{i}} \right)} - {I(x)}} \right)^{2}}}} & (2)\end{matrix}$

The calculation of the degree of non-focus J is not limited to themethod using the variance of Equation (2). For example, an average valueof differences from the average pixel values of the pixels whosepositions accord with each other when superimposed, which is expressedby Equation (3), may be calculated as the degree of non-focus J.

$\begin{matrix}{{J(x)} = {\frac{1}{{W(x)}}{\sum\limits_{i \in {W{(x)}}}^{\;}{{{M_{i}\left( {x - o_{i}} \right)} - {I(x)}}}}}} & (3)\end{matrix}$

With regard to a region of the microlens image 131 in which the focusedobject is imaged, the images are expanded, shifted, and superimposed soas to accord with each other. Thus, the variation in the pixel values ofthe pixels of the superimposed portion of the microlens images Mi, whosepositions accord with each other when superimposed, is small. Therefore,the value of the degree of non-focus J calculated by using Equation (2)or Equation (3) described above becomes small. On the other hand, withregard to a region in which the non-focused object is imaged, even whenthe expanded microlens images M_(i) are superimposed, the images do notaccord with each other. Therefore, since the variation in pixel valuesbecomes larger, the value of the degree of non-focus J calculated byusing Equation (2) or Equation (3) becomes larger.

The example has been described above in which the non-focus degreecalculator 103 calculates the degree of non-focus J by generating therefocused image 140 as in the refocused image generator 102, but theinvention is not limited to this example. That is, the non-focus degreecalculator 103 may receive an input of information regarding aprocessing procedure of the refocused image generator 102 and calculatethe degree of non-focus J. For example, as an image processing device100′ is exemplified in FIG. 9, the non-focus degree calculator 103 mayacquire the expanded microlens images M_(i) generated in the processingprocedure performed by the refocused image generator 102 and calculatethe degree of non-focus J using the microlens images M_(i).

In the case where the degree of non-focus J is calculated from thecompound eye image 130 captured and acquired by the camera array, avariation in the pixel values of the images obtained by shifting theunit images by an amount of shift corresponding to an amount shifted bythe refocused image generator 102 and superimposing the unit images iscalculated.

The method of calculating the degree of non-focus is not limited to theabove-described method. For example, distance information indicating adistance between an object and the acquisition unit 101 (the main lens110) may be acquired separately and the degree of non-focus J may beobtained based on the distance information. In this case, the acquireddistance information is calculated such that the distance thus acquiredis further apart from a distance that has been focused at by therefocused image generator 102, the degree of non-focus J is larger. Thedistance information can be calculated from information separatelymeasured by a distance sensor or a stereo camera. Further, the distanceinformation may be acquired by calculating the amount of shift of theunit image in the compound eye image.

Next, the blurring processing of step S24 in the flowchart of FIG. 4will be described. The blurring processor 104 performs the blurringprocessing on the refocused image 140 generated in step S22 by therefocused image generator 102 based on the degree of non-focus Jcalculated in step S23 by the non-focus degree calculator 103.

FIG. 10 is a flowchart illustrating an example of the blurringprocessing performed by the blurring processor 104. In step S1101, theblurring processor 104 selects single coordinates x_(p) on the refocusedimage 140 supplied from the refocused image generator 102 as thecoordinates of a pixel of interest. Next, in step S1102, the blurringprocessor 104 determines the size of a kernel used to calculate thepixel value on an output image according to the degree of non-focus J(x_(p)) of the coordinates x_(p). The kernel is an image range in whicha filtering processing is performed on single coordinates (pixel ofinterest) at the time when an image is blurred through the filteringprocessing. That is, the kernel can be said to be a range containingneighboring pixels located near the pixel of interest.

For example, the blurring processor 104 enlarges the size of the kernelas the degree of non-focus J (x_(p)) of the coordinates x_(p) is larger.On the assumption that k (x_(p)) is the size of the kernel at thecoordinates x_(p), a size k (k_(p)) of the kernel is calculated byEquation (4) below. In Equation (4), a value α is a positive integer.

k(x _(p))=αJ(x _(p))  (4)

The blurring processor 104 may set the upper and lower limit values ofthe size k(x_(p)) of the kernel. FIG. 11 is a diagram illustrating anexample in which the upper and lower limit values are set in the sizek(x_(p)) of the kernel. The blurring processor 104 calculates the sizek(x_(p)) of the kernel from the degree of non-focus J(x_(p)) accordingto a relation illustrated in FIG. 11.

More specifically, the blurring processor 104 sets the size k(x_(p)) ofthe kernel to a value k₁, when the value of the degree of non-focusJ(x_(p)) is in the range from 0 to the lower limit value th₁. Theblurring processor 104 sets the size k(x_(p)) of the kernel so as togradually increase from the value k₁ to a value k₂ according to thevalue of the degree of non-focus J(x_(p)), when the value of the degreeof non-focus J(x_(p)) is in the range of the lower limit value th₁ tothe upper limit value th₂. The blurring processor 104 sets the value ofthe size k(x_(p)) of the kernel to the upper limit value k₂, when thevalue of the degree of non-focus J(x_(p)) exceeds the upper limit valueth₂.

A relation between the degree of non-focus J(x_(p)) and the sizek(x_(p)) of the kernel may be obtained through calculation, asillustrated in FIG. 11, or may be prepared in advance as a table.

When the value of the size k(x_(p)) of the kernel is quantized so thatan odd integer is equal to or greater than 1, the calculation ofsubsequent step S1103 can be preferably simplified.

The example has been described above in which the size k(x_(p)) of thekernel is determined only based on the degree of non-focus J(x_(p)) atsingle coordinates x_(p), but the invention is not limited to thisexample. For example, when the size k(x_(p)) of the kernel is determinedusing the value of the degree of non-focus J of the coordinates near thecoordinates x_(p), artifact of the refocused image 140 can preferably bereduced more efficiently.

For example, the degree of non-focus J of each coordinate of arectangular region in a range of ±b pixels set in the horizontal andvertical directions centering on the coordinates x_(p) is obtained, themaximum value of each of the calculated degrees of non-focus J isobtained, and the maximum value is set as the degree of non-focus J′(x_(p)). The size k(x_(p)) of the kernel may be calculated using thedegree of non-focus J′ (x_(p)), instead of the degree of non-focusJ(x_(p)) in Equation (4) described above. For example, the size k(x_(p))of the kernel is calculated by Equation (5) below.

k(x _(p))=αJ′(x _(p))  (5)

The method of calculating the above-described degree of non-focus J′(x_(p)) is not limited to the method in Equation (5). For example, thesize k(x_(p)) of the kernel can also be calculated using an averagevalue of the degrees of non-focus J of the coordinates of therectangular region in the range of ±b pixels set in the horizontal andvertical directions centering on the coordinates x_(p).

Next, in step S1103, the blurring processor 104 calculates a pixel valueI′ (x_(p)) of an output image at the coordinates x_(p) by Equation (6)below using the size k(x_(p)) of the kernel obtained in step S1102.

$\begin{matrix}{{I^{\prime}\left( x_{p} \right)} = {\frac{1}{{k\left( x_{p} \right)}^{2}}{\sum\limits_{x \in {Y{(x_{p})}}}^{\;}\; {I(x)}}}} & (6)\end{matrix}$

In Equation (6), it is assumed that Y(x_(p)) is a set of the coordinateswithin the rectangular region of which lengths in the horizontal andvertical directions centering on the coordinates x_(p) are the sizek(x_(p)) of the kernel. A pixel value I(x) indicates the pixel value ofeach pixel within the kernel. An average value of the pixel valueswithin the rectangular region is set as the pixel value I′ (x_(p)). Thepixel value I′ (x_(p)) is a pixel value obtained by performing theblurring processing on the pixels of the refocused image 140 suppliedfrom the refocused image generator 102.

The blurring processing performed by the blurring processor 104 is notlimited to the above-described method. For example, a pixel valueI″(x_(p)) obtained by performing the blurring processing on the pixelhaving the pixel value I(x_(p)) can also be calculated using a Gaussiankernel determining a variance by the value of the size k(x_(p)) of thekernel.

The example has been described above in which the blurring processing isperformed sequentially on the pixel values of the respective coordinateswithin the refocused image 140, but the embodiment is not limited tothis example. For example, a map of the degree of non-focus J may begenerated for the refocused image 140 and the blurring processing may beperformed based on this map.

FIG. 12 is a diagram illustrating an example of the map of the degree ofnon-focus J. In FIG. 12, as the value of the degree of non-focus Jdecreases, denser hatching is given and illustrated. In the example ofFIG. 12, the refocused image 140 includes images 150, 151, and 152 ofthree objects whose distances from the acquisition unit 101 (the mainlens 110) are different from each other. The object of the image 150 isthe closest to the acquisition unit 101 and the object of the image 152is the remotest from the acquisition unit 101.

The refocused image 140 is assumed to be generated to be focused on theobject (the image 151) located at a middle distance among the objects ofthe images 150, 151, and 152. Accordingly, in the refocused image 140,the degree of non-focus J is a small value at coordinates included inthe image 151. In the example of FIG. 12, the degree of non-focus J isthe second small value at coordinates included in the image 150 and isthe third small value at coordinates included in the image 152. In theexample of FIG. 12, in the refocused image 140, the degree of non-focusJ is a larger value in a region other than the images 150, 151, and 152than in the images 150, 151, and 152.

The blurring processor 104 performs the blurring processing on therefocused image using a map obtained by performing a filteringprocessing on the map of the degree of non-focus J. For example, adilation filtering processing can be used as the filtering processing.The invention is not limited to this configuration. A filteringprocessing using a general blurring processing filter (an average valuefilter or the like) may be performed on this map.

Next, in step S1104, the blurring processor 104 determines whether theprocessing of step S1101 to step S1103 is performed on all of thecoordinates in the refocused image 140 supplied from the refocused imagegenerator 102. When the blurring processor 104 determines the processingis not performed on all of the coordinates, the blurring processor 104returns the process to step S1101, select single coordinates notsubjected to the blurring processing in the refocused image 140, andperforms the blurring processing on the selected coordinates.

Conversely, when the blurring processor 104 determines that theprocessing of step S1101 to step S1103 is performed on all of thecoordinates in the refocused image 140, the blurring processor 104outputs the pixel values I′ of all the coordinates of the refocusedimage 140 as an output image subjected to the blurring processing.

Thus, the image processing device 100 according to the first embodimentis configured to blur a region with the large degree of non-focus J,that is, a non-focused region of the refocused image 140 moreintensively. The image processing device 100 can perform this processingto generate the refocused image in which the artifact is suppressed.

The example has been described above in which the compound eye image 130is acquired every time by the acquisition unit 101 and is input to theimage processing device 100, but the invention is not limited to thisexample. For example, the acquisition unit 101 may be configured as astorage device such as a hard disk, the separately acquired compound eyeimage 130 may be stored and accumulated in the storage device such as ahard disk, and the compound eye image 130 stored in the storage devicemay be input to the image processing device 100.

Second Embodiment

Next, a second embodiment will be described. The second embodiment is anexample in which the image processing device 100 according to the firstembodiment includes an optical system and is applied to an imagingdevice capable of storing and displaying an output image.

FIG. 13 is a diagram illustrating example of the configuration of animaging device 200 according to the second embodiment. In FIG. 13, thesame reference numerals are given to constituent elements common tothose described above in FIG. 1 and the detailed description thereofwill not be repeated. As exemplified in FIG. 13, the imaging device 200includes an imaging unit 160, an image processing device 100, anoperation unit 210, a memory 211, and a display unit 212.

The entire process of the imaging device 200 is controlled according toa program by a CPU (not illustrated). The imaging unit 160 includes theoptical system exemplified in FIG. 2 and a sensor 112 in correspondencewith the above-described acquisition unit 101.

The memory 211 is, for example, a non-volatile semiconductor memory andstores an output image output from the image processing device 100. Thedisplay unit 212 includes a display device such as an LCD (LiquidCrystal Display) and a driving circuit that drives the display device.The display unit 212 displays the output image output from the imageprocessing device 100.

The operation unit 210 receives a user's input. For example, a distancethat the user wants to be focused at for the refocused image 140 can bedesignated in the image processing device 100 through the user's inputon the operation unit 210. The distance that the user wants to befocused at is, for example, has a value according to the expansionmagnification m when the unit images are expanded. The operation unit210 can receive a user's input or the like of an imaging timing of theimaging unit 160, a storage timing of the output image in the memory211, and focusing control at the time of imaging.

In this configuration, the imaging device 200 designates the focusingdistance at the time of imaging according to a user's input on theoperation unit 210. The imaging device 200 designates a timing at whichthe compound eye image 130 output from the imaging unit 160 is acquiredin the image processing device 100 according to a user's input on theoperation unit 210.

The imaging device 200 causes the display unit 212 to display an outputimage subjected even to the blurring processing by the blurringprocessor 104 according to the focusing distance for the refocused image140 designated by a user's input on the operation unit 210. For example,the user can re-input the focusing distance from the operation unit 210with reference to display of the display unit 212. For example, when theuser obtains an interesting output image, the user operates theoperation unit 210 to store the output image in the memory 211.

Thus, the imaging device 200 according to the second embodiment obtainsthe degree of non-focus J of the refocused image 140 generated from thecompound eye image 130 captured by the imaging unit 160. In a regionwhich is not in focus and has the larger degree of non-focus J, anoutput image obtained by blurring the refocused image 140 moreintensively is obtained. Therefore, the user can generate the refocusedimage in which the artifact is suppressed from the compound eye image130 captured by the imaging device 200 and obtains the refocused imageas the output image.

Third Embodiment

Next, a third embodiment will be described. The third embodiment is anexample in which the image processing device 100 according to the firstembodiment is applied to a sensor device that includes an optical systemand is configured to transmit an output image to the outside and receivean operation signal from the outside.

FIG. 14 is a diagram illustrating an example of the configuration of asensor device 300 according to the third embodiment. In FIG. 14, thesame reference numerals are given to constituent elements common tothose described above in FIGS. 1 and 13 and the detailed descriptionthereof will not be repeated. As exemplified in FIG. 14, the sensordevice 300 includes an imaging unit 160 and an image processing device100.

An operation signal transmitted from the outside through wired orwireless communication is received by the sensor device 300 and is inputto the image processing device 100. An output image output from theimage processing device 100 is output from the sensor device 300 throughwired or wireless communication.

In this configuration, the sensor device 300 generates an output imagesubjected even to the blurring processing by the blurring processor 104according to the focusing distance for the refocused image designated bythe operation signal transmitted from the outside. The output image istransmitted from the sensor device 300 to the outside. In the outside,the received output image can be displayed and an operation signalconfigured to designate a focused position can also be transmitted tothe sensor device 300 based on the display. A re-input from theoperation unit 210 can be performed.

The sensor device 300 can be applied to, for example, a monitoringcamera. In this case, display is monitored using an output image fromthe sensor device 300 located at a remote place. When the display imageincludes a doubtful image, a focusing distance for the doubtful imageportion is designated and an operation signal is transmitted to thesensor device 300. The sensor device 300 regenerates the refocused image140 in response to the operation signal, performs the blurringprocessing, and transmits an output image. The details of the doubtfulimage portion can be confirmed using the output image on which thefocusing distance is reset and the artifact is suppressed.

Fourth Embodiment

Next, a fourth embodiment will be described. The fourth embodiment is anexample of an image processing system in which the image processingdevice 100 according to the first embodiment is constructed on a networkcloud. FIG. 15 is a diagram illustrating an example of the configurationof the image processing system according to the fourth embodiment. InFIG. 15, the same reference numerals are given to constituent elementscommon to those described above in FIG. 1 and the detailed descriptionthereof will not be repeated.

In FIG. 15, in the image processing system, the image processing device100 is constructed on a network cloud 500. The network cloud 500 is anetwork group that includes a plurality of computers connected to eachother in a network and displays only input and output as a black box ofwhich the inside is hidden from the outside. The network cloud 500 isassumed to use, for example, TCP/IP (Transmission ControlProtocol/Internet Protocol) as a communication protocol.

The compound eye image 130 acquired by the acquisition unit 101 istransmitted to the network cloud 500 via the communication unit 510 andis input to the image processing device 100. The compound eye image 130transmitted via the communication unit 510 may be accumulated and storedin a server device or the like on the network cloud 500. The imageprocessing device 100 generates the refocused image 140 based on thecompound eye image 130 transmitted via the communication unit 510,calculates the degree of non-focus J and generates an output image byperforming the blurring processing on the refocused image 140 based onthe degree of non-focus J.

The generated output image is output from the image processing device100 and, for example, a terminal device 511 which is a PC (PersonalComputer) receives the output image from the network cloud 500. Theterminal device 511 can display the received output image on a displayand transmit an operation signal configured to designate a focusingdistance in response to a user's input to the network cloud 500. Theimage processing device 100 regenerates the refocused image 140 based onthe designated focusing distance in response to the operation signal andgenerates an output image by performing the blurring processing. Theoutput image is retransmitted from the network cloud 500 to the terminaldevice 511.

According to the fourth embodiment, the user can obtain the output imagewhich is generated by the image processing device 100 and in which theartifact is suppressed through the blurring processing, even when theuser does not possess the image processing device 100.

Another Embodiment

The image processing device 100 according to the above-describedembodiments may be realized using a general computer device as basichardware. FIG. 16 is a diagram illustrating an example of theconfiguration of a computer device 400 to which the image processingdevice 100 can be applied according to another embodiment.

In the computer device 400 exemplified in FIG. 16, a CPU (CentralProcessing Unit) 402, a ROM (Read Only Memory) 403, a RAM (Random AccessMemory) 404, and a display control unit 405 are connected to a bus 401.A storage 407, a drive device 408, an input unit 409, a communicationI/F 410, and a camera I/F 420 are also connected to the bus 401. Thestorage 407 is a storage medium capable of storing data in anon-volatile manner and is, for example, a hard disk. The invention isnot limited this configuration, but the storage 407 may be anon-volatile semiconductor memory such as a flash memory.

The CPU 402 controls the entire computer device 400 using the RAM 404 asa work memory according to programs stored in the ROM 403 and thestorage 407. The display control unit 405 converts a display controlsignal generated by the CPU 402 into a signal which the display unit 406can display and outputs the converted signal.

The storage 407 stores a program executed by the above-described CPU 402or various kinds of data. A detachable recording medium (notillustrated) can be mounted on the drive device 408, and thus the drivedevice 408 can read and write data from and on the recording medium.Examples of the recording medium treated by the drive device 408 includea disk recording medium such as a CD (Compact Disk) or a DVD (DigitalVersatile Disk) and a non-volatile semiconductor memory.

The input unit 409 inputs data from the outside. For example, the inputunit 409 includes a predetermined interface such as a USB (UniversalSerial Bus) or IEEE 1394 (Institute of Electrical and ElectronicsEngineers 1394) and inputs data from an external device through theinterface. Image data of an input image can be input from the input unit409.

An input device such as a keyboard or a mouse receiving a user's inputis connected to the input unit 409. For example, a user can give aninstruction to the computer device 400 by operating the input deviceaccording to display of the display unit 406. The input device receivinga user's input may be configured to be integrated with the display unit406. At this time, the input device may be preferably configured as atouch panel that outputs a control signal according to a pressedposition and transmits an image of the display unit 406.

The communication I/F 410 performs communication with an externalcommunication network using a predetermined protocol.

The camera I/F 420 is an interface between the acquisition unit 101 andthe computer device 400. The compound eye image 130 acquired by theacquisition unit 101 is received via the camera I/F 420 by the computerdevice 400 and is stored in, for example, the RAM 404 or the storage407. The camera I/F 420 can supply a control signal to the acquisitionunit 101 in response to a command of the CPU 402.

The refocused image generator 102, the non-focus degree calculator 103,and the blurring processor 104 described above are realized by an imageprocessing program operating on the CPU 402. The image processingprogram configured to execute image processing according to theembodiments is recorded as a file of an installable format or anexecutable format in a computer-readable recording medium such as a CDor a DVD to be supplied. The invention is not limited to thisconfiguration, but the image processing program may be stored in advancein the ROM 403 to be supplied.

The image processing program configured to execute the image processingaccording to the embodiments may be stored in a computer connected to acommunication network such as the Internet and may be downloaded via thecommunication network to be supplied. The image processing programconfigured to execute the image processing according to the embodimentsmay be supplied or distributed via a communication network such as theInternet.

For example, the image processing program configured to execute theimage processing according to the embodiments is designed to have amodule structure including the above-described units (the refocusedimage generator 102, the non-focus degree calculator 103, and theblurring processor 104). Therefore, for example, the CPU 402 as actualhardware reads the image processing program from the storage 407 andexecutes the image processing program, and thus the above-describedunits are loaded on a main storage device (for example, the RAM 404) sothat the units are generated on the main storage device.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. An image processing device comprising: agenerator configured to generate, from a plurality of unit images inwhich points on an object are imaged by an imaging unit at differentpositions according to distances between the imaging unit and thepositions of the points on the object, a refocused image focused at apredetermined distance; and a processor configured to perform blurringprocessing on each pixel of the refocused image according to anintensity corresponding to a focusing degree of the pixel of therefocused image.
 2. The device according to claim 1, further comprising:a calculator configured to calculate a defocusing degree indicating alarger value as the focusing degree is lower for each pixel of therefocused image, wherein the processor performs the blurring processingon each pixel of the refocused image more intensively as the defocusingdegree of the pixel of the refocused image is larger.
 3. The deviceaccording to claim 2, wherein the processor performs the blurringprocessing on a pixel of interest of the refocused image with a blurintensity corresponding to a magnitude of each defocusing degreecalculated for each of neighboring pixels located near the pixel ofinterest by calculator.
 4. The device according to claim 2, wherein theprocessor performs the blurring processing on a pixel of interest of therefocused image using a pixel value of a pixel within a range that isset corresponding to a magnitude of each defocusing degree calculatedfor each of neighboring pixels located near the pixel of interest by thecalculator.
 5. The device according to claim 2, wherein the processorperforms the blurring processing on a pixel of interest with a blurintensity corresponding to a magnitude of the maximum value of thedefocusing degree for a pixel within a predetermined range including thepixel of interest.
 6. The device according to claim 2, wherein theprocessor performs the blurring processing on a pixel of interest usinga pixel value of a pixel within a range that is set corresponding to amagnitude of the maximum value of the defocusing degree for a pixelwithin a predetermined range including the pixel of interest.
 7. Thedevice according to claim 2, wherein the calculator calculates thedefocusing degree with a magnitude corresponding to a variation in pixelvalues of the pixels whose pixel positions accord with each other in theunit images that are expanded at an expansion ratio determined accordingto the predetermined distance and are superimposed.
 8. The deviceaccording to claim 2, wherein the calculator calculates the defocusingdegree with a magnitude corresponding to a variation in pixel values ofthe pixels whose pixel positions accord with each other in the unitimages that are shifted by an amount of shift determined according tothe predetermined distance and are superimposed.
 9. The device accordingto claim 7, wherein the variation is a variance of the pixel values ofthe pixels of which the pixel positions accord with each other.
 10. Thedevice according to claim 8, wherein the variation is a variance of thepixel values of the pixels of which the pixel positions accord with eachother.
 11. The device according to claim 2, further comprising: adistance acquisition unit configured to acquire a distance between theimaging unit and the object, wherein the calculator calculates thedefocusing degree with a magnitude corresponding to a difference betweenthe distance acquired by the distance acquisition unit and thepredetermined distance.
 12. The device according to claim 1, furthercomprising: the imaging unit; a reception unit configured to receiveinformation that indicates at least the predetermined distance and istransmitted from the outside; and a transmission unit configured totransmit the refocused image subjected to the blurring processing by theprocessor to the outside.
 13. The device according to claim 1, furthercomprising: the imaging unit; an input unit configured to receive auser's input of information indicating at least the predetermineddistance; and a display unit configured to display the refocused imagesubjected to the blurring processing by the processor.
 14. An imageprocessing method comprising: generating, from a plurality of unitimages in which points on an object are imaged by an imaging unit atdifferent positions according to distances between the imaging unit andthe positions of the points on the object, a refocused image focused ata predetermined distance; and performing blurring processing on eachpixel of the refocused image according to an intensity corresponding toa focusing degree of the pixel of the refocused image.
 15. A computerprogram product comprising a computer-readable medium containing acomputer program that causes a computer to perform: generating, from aplurality of unit images in which points on an object are imaged by animaging unit at different positions according to distances between theimaging unit and the positions of the points on the object, a refocusedimage focused at a predetermined distance; and performing blurringprocessing on each pixel of the refocused image according to anintensity corresponding to a focusing degree of the pixel of therefocused image.